NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension
نویسندگان
چکیده
One-shot neural architecture search (NAS) substantially improves the efficiency by training one supernet to estimate performance of every possible child (i.e., subnet). However, inconsistency characteristics among subnets incurs serious interference in optimization, resulting poor ranking correlation subnets. Subsequent explorations decompose weights via a particular criterion, e.g., gradient matching, reduce interference; yet they suffer from huge computational cost and low space separability. In this work, we propose lightweight effective local intrinsic dimension (LID)-based method NAS-LID. NAS-LID evaluates geometrical properties architectures calculating low-cost LID features layer-by-layer, similarity characterized enjoys better separability compared with gradients, which thus effectively reduces Extensive experiments on NASBench-201 indicate that achieves superior efficiency. Specifically, gradient-driven method, can save up 86% GPU memory overhead when searching NASBench-201. We also demonstrate effectiveness ProxylessNAS OFA spaces. Source code:https://github.com/marsggbo/NAS-LID.
منابع مشابه
Efficient Neural Architecture Search via Parameter Sharing
We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller discovers neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on a validation set. Meanwhile the model cor...
متن کاملNeural networks for estimating intrinsic dimension.
We consider the problem of feature extraction and determination of intrinsic dimensionality of observation data. One of the common approaches to this problem is to use autoassociative neural networks with a "bottleneck" projecting layer. We propose a different approach in which a neural network performs a topological mapping that creates a nonlinear lower-dimensional projection of the data. The...
متن کاملSimple And Efficient Architecture Search for Convolutional Neural Networks
Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by sho...
متن کاملNeural Networks with Finite Intrinsic Dimension have no Spurious Valleys
Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may be responsible for such success. In particular, several authors have noted that overparametriza...
متن کاملProgressive Neural Architecture Search
We propose a method for learning CNN structures that is more efficient than previous approaches: instead of using reinforcement learning (RL) or genetic algorithms (GA), we use a sequential model-based optimization (SMBO) strategy, in which we search for architectures in order of increasing complexity, while simultaneously learning a surrogate function to guide the search, similar to A* search....
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Proceedings of the ... AAAI Conference on Artificial Intelligence
سال: 2023
ISSN: ['2159-5399', '2374-3468']
DOI: https://doi.org/10.1609/aaai.v37i6.25949